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It is only by the considered and judicious use of a range of 
methodologies that educators, policymakers and the public can make real 
advances to the system for teaching students to reach their full reading 
potential. By limiting reading instruction research to experimental and 
quasi experimental studies, the National Reading Panel missed critical areas 
included in qualitative and ethnographic research and nullified the 
possibility of solving learning and teaching problems. Language and cognitive 
differences, differential access to books, and inequalities in schooling are 
possible causes for differential achievement rates, a type of educational 
research addressed through ethnography. The rigor represented in codified 
ethnographic research methods produces scientifically valid and reliable 
data, as do descriptive and correlational methodologies and their 
combinations. Ethnographic, correlational, and descriptive methodologies 
demonstrate a value added nature which increases their potential contribution 
to educational research. These methodologies are needed to allow 
investigation of insights, possible operative factors, and new information, 
which need to be understood before limiting studies to experimental, 
hypothesis-testing approaches. (EF) 
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In spring 1998, the director of the National Institute of Child Health and Human Development appointed the 
National Reading Panel (NRPI to carry out a request by the United States Congress to study the 
effectiveness of various approaches to teaching children how to read. The panel subsequently defined the 
procedures by which it would conduct its business and the questions that it would use to guide its study. 

The panel has now published its results, available at 
www.nationalreadinaDanel.orQ/Documents/default.htm . 

The National Reading Conference fNRC) . a professional organization dedicated to literacy research, created 
a committee to respond to the NRP report by addressing the following questions: 

1. Did the panel do what it said it would do to answer the questions it had posed? 

2. What, if anything, did the panel leave out within the questions it created and in defining the 
questions themselves? 

3. What research traditions, if any, have not been considered in the panel's work? 

Following is the first in a planned series of commentaries prepared by members of the NRC committee. It 
presents one response to the third question. 

Martha Rapp Ruddell 

Chair, National Reading Conference Committee to Respond to the National Reading Panel 

Past President, National Reading Conference 



The Role of Qualitative and Ethnographic Research in Educational Policy 

Victoria Purcell-Gates 



The National Reading Panel, charged with identifying research that could inform policy and practice in reading, 
has now concluded its exhaustive review. The panel focused its review on experimental and quasiexperimental 
research which, by necessity, meant that it focused on a small set of questions. I say "by necessity" because 
the areas of instruction in reading that have been examined by experimental or quasiexperimental research 
are of themselves few and far between, although certainly of central importance to the learning-to-read 
process. 

The point I wish to argue here is that by limiting the type of research used to address the issue of how to 
guide policy and practice in reading instruction to experimental and quasiexperimental studies, the panel 
missed critical areas that have been examined by qualitative and ethnographic research. Further, a clear 
danger resulting from this limitation to particular types of methodologies is the establishment in the minds of 
funders and users of research that only experimental and quasiexperimental research provide us with answers 
to educational problems. A second theme of this commentary, then, is my view that we do not stand a chance 
of solving problems of learning and teaching if we confine ourselves solely to these methodologies. Rather, it 
is only by the considered and judicious use of a range of methodologies, rigorously applied and intelligently 
synthesized, that we as educators, policy makers, and the informed and concerned public can hope to make 
real advances and reforms to our system and in our procedures for educating all our students to their full 
potentials. 



Proving, Informing, or Both? 

There is no doubt that experimental, and to some degree quasiexperimental, research is required to "prove" 
the effectiveness of an instructional approach, method, or intervention. This to some degree explains the 
panel's exclusion of other methodologies in its research review (see, e.g., Shanahan, 1999 ; online document ). 
Identifying the effects of specified instructional techniques can only be accomplished with carefully designed 
studies that employ random sampling, random assignment to condition, control groups, large Ns, and analysis 
techniques that allow for the accounting of the myriad confounding variables that are inevitable in the study of 
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socially situated activities such as teaching and learning. Generalization from such studies is bounded and 
constrained by descriptions of subjects, teachers, teacher training in the technique under study, fidelity of 
intervention implementation, and evidence of sustainability and scalability. 

Clearly, this ideal, requisite though it may be, is difficult to attain, expensive to accomplish, and limited to 
answering only those questions that can actually be examined with such methodologies ( Boruch, 1999 , & 
Gueron, 1999. single online document ). Gueron, drawing on her experience with research on training 
programs and welfare reform, lists eight criteria useful forjudging when random assignment is the right tool: 

1. When the key question is "Does the program or reform make a difference?" (questions of program 
impact) 

2. When the program under study is sufficiently different from business as usual and you can maintain that 
distinction over time 

3. When the research does not denying people access to an entitlement 

4. When the researchers are addressing an important or unanswered question 

5. When adequate procedures are in place to inform program participants about the study and to ensure 
confidentiality of data 

6. When there is no easier way to get a good answer 

7. When you can get cooperation 

8. When you have the resources and ability to do a quality study 

While experimental and quasiexperimental studies are the gold standard for examining program impact, there 
are many critical issues facing education and educators that raise other types of questions. For example, most 
reasonable people would probably agree that a central concern in the field of education is that of differential 
achievement rates. Throughout the history of public education, children and adults from nonmainstream 
groups or for low socioeconomic status (SES) have as a whole failed to achieve academic success to the same 
levels and at the same rate as learners from mainstream, sociopolitically empowered groups ( Kaestle, 
Damon-Moore. Stedman. Tinslev. &Trollinaer. iggi T Descriptive and correlational studies (both types of 
qualitative research) repeatedly have confirmed this disturbing pattern. Other studies using qualitative and 
quantitative research methods have examined possible causes as subcorrelates — language differences, 
cognitive differences, differential access to books, inequalities in schooling, and so on. Clearly, without the 
ability, or right, to control probable confounding variables or randomly assign to condition (of socioeconomic 
status?), we can never establish true causation. However, we can gain real insight into the issue by employing 
nonexperimental methods. Further, with judicious and careful synthesis, we can come to understand the 
landscape of nonmainstream status and educational achievement to such a degree that we may be able to 
design experimental studies that examine the impact of educational programs on this SES-related 
achievement gap. 

Many of our pressing and unanswered educational questions are of this type -- deeply embedded in issues of 
race, culture, class, gender, and family income. For most of these questions, we do not possess sufficient 
information or insights, gained through systematic, rigorous, nonexperimental research, to design reasonable 
experimental interventions and research studies that meet the criteria set forth by Gueron (1999) , listed 
above. 

LeComote and Schensul fl999. p. 2^ describe ethnography, one type of nonexperimental research that uses 
both qualitative and quantitative data, as a research method "designed for discovery." Ethnography, they 
point out, is particularly suited to gaining insight into questions embedded in social and cultural communities 
and practices such as education. It is scientific, investigative, uses the researcher as the primary tool of data 
collection, employs rigorous research methods and data collection techniques to avoid bias and ensure 
accuracy of data, emphasizes and builds on the perspectives of the people in the research setting, and is 
inductive, building local theories for testing and adapting for use both locally and elsewhere. 

The rigor represented in codified ethnographic research methods produces scientifically valid and reliable data. 
The same is true for other nonexperimental research designs using qualitative and quantitative methods -- 
descriptive and correlational methodologies, and different combinations of these. Each type of research has its 
own criteria for rigor, and each answers different types of questions. Thus, all types of research, from 
experimental to ethnographic, provide us with important lenses through which to study and examine critical 
educational questions and issues. Because they are different, each type of research provides us with unique 
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answers and insights, and each type of research presents its own limitations affecting generalizability, types of 
questions it can address, and the nature of the insights it can offer. 



Nonexperimental Research Methods That Inform Policy and Practice 

Following is a brief listing of nonexperimental research methodologies that use qualitative and quantitative 
methods, along with brief descriptions of the types of questions they answer and the considerations 
researchers and the public should keep in mind when interpreting and applying their results. The purpose of 
this listing is to demonstrate the "value added" nature of these methodologies in relation to their potential 
contribution to educational research that can inform policy and practice. Note that this is intended only as a 
quick description of nonexperimental methodologies; within each category there are many different 
approaches and standards in relation to researcher-researched relationships, stances, and procedures. 

The table below relies heavily on two main sources: Understanding Research in Reading and Writing ( Kamil. 
Lanqer, & Shanahan, 1985) and Designing and Conducting Ethnopraphic Research (LeComote & <^chensul. 
1999T 



Type of Research 


Purposes/Uses of Research 


Issues for Interpretation 


Correlational 


To determine relations among variables 
that are hard or impossible to 
manipulate 

To determine the reliability of testing 
instruments 

To determine the interrelationships 
among a number of dependent 
variables in a given situation 


Correlations cannot reveal definitive 
causation. 

Effect size is more important than 
simple correlations. 

High intercorrelations among multiple 
measures indicate they are measuring 
the same thing. 


Descriptive 


To describe characteristics, properties, 
or relationships of groups, events, or 
phenomena without manipulation of the 
phenomena under study 

Especially useful to increase awareness 
of particular problems or to provide 
baseline data for future studies 


Results cannot establish causation. 

Instruments (surveys, interviews, 
observations, tests, etc.) must be 
appropriate for the questions asked. 

Instruments must have established 
validity and reliability. 

Because these studies begin with 
categories of interest already 
established, they cannot discover 
factors of potential importance and 
interest. 


Ethnographic 


To define a problem when the problem 
is not clear 

To define a problem that is complex 
and embedded in multiple systems or 
sectors 

To identify participants when the 
participants, sectors, or stakeholders 
are not yet known or identified 


Research methods must be chosen to 
provide appropriate information about 
the problem being studied. 

Data must be systematically collected 
and recursively analyzed. 

Conclusions are applicable primarily to 
the setting under investigation; 
application to other settings relies on 
the degree to which the contexts are 
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To clarify the range of settings where 
the problem or situation occurs at times 
when the settings are not fully 
identified, known, or understood 

To explore the factors associated with 
the problem in order to understand it 



similar along all relevant dimensions. 

Relationships uncovered cannot be 
interpreted as causative. 

Conclusions can be used primarily to 
provide insights, to contribute to 
theory, and to suggest factors to 
consider for other types of studies. 



The Need for More Than Experiments 

Many educational issues and problems clearly require research that draws on multiple perspectives, 
approaches, and procedures. We simply do not know enough about the complexities of the complex process of 
learning in schools to design effective experimental studies that will solve all of our problems. We need 
methodologies that will allow us to probe for insights, for possible operative factors, for new information, 
before we can begin to think about limiting our research studies to experimental, hypothesis-testing 
approaches. 

On December 2, 1999, a discussion group appointed by the National Reading Conference formulated a sample 
of questions that have been addressed by nonexperimental research. The group consisted of Yetta Goodman, 
Judith Green, Jerome Harste, Steven Stahl, Jeffrey Wood, and myself. We offer this sample by way of 
illustrating the questions, which we consider key to understanding how children learn to read and write in 
schools, that have been effectively investigated by research using other than experimental and 
quasiexperimental methodologies. 

1. What resources in terms of experiences, language, cognition, and social and cultural expectations 
brought to school by learners do they then rely on to interpret and learn from instruction? How do these 
resources influence how learners "take" from instruction? 

2. What opportunities exist for learning, at both classroom and school levels, for different children in 
different school settings? 

3. What are the relationships between writing development and reading development? 

4. How do different instructional practices affect instructional outcomes differentially? 

5. What knowledge do teachers draw upon or use to inform their practice? 

6. What are some relationships between classroom talk and learning to read? 

7. What different demands do learners face when reading in different disciplines, such as science, math, or 
social studies? 

8. How do social interactions among students influence their learning to read? 

9. How do different instructional contexts — such as teacher-directed group reading, reading to answer 
questions, or reading on one's own — influence the strategies readers use to process print for meaning? 

10. What are some relationships between individual differences and instructional outcomes? Do these 
change in different instructional contexts? 

Insights gained from research into these questions are particularly informative, not only to guide development 
of future experimental studies, but also to present-day considerations of instructional policy. The answers we 
have gleaned provide the texture for teachers who are considering what research says to them about how to 
teach each individual child in the classroom in ways that will result in personal growth and achievement for 
that child. Research into questions such as these helps teachers understand, or at least begin to 
problem-solve, the inevitable failures with specific children of strategies documented as successful overall in 
large, generalizable experimental studies. 



A Word to Policy Makers and Funding Agencies 

This commentary is intended to urge policy makers and funders of research to acknowledge the contributions 
that nonexperimental research can make to decisions regarding instructional policy as well as to building our 
knowledge base of learning and teaching. I urge you not to restrict educational research to one or two 
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paradigms. To do so would unacceptably restrict our ability to explore critical issues, insights about which are 
needed before we can begin to address the complex challenges of educating all learners in this complex, 
multicultural society of ours to their full potentials. I urge you to acknowledge fully and publicly the need for 
different lenses to address these challenges. At the same time, I urge you to insist on high research standard: 
within each research paradigm, standards for rigor that will guarantee results that can be trusted and relied 
upon as they are used, according to their unique purposes, by policy makers, funders of research, school 
administrators, and individual teachers. 



References 

Boruch, R. (1999). Can we make education policy on the basis of evidence? What constitutes high quality 
education research and how can it be incorporated into policymaking? (transcript of a Brookings Press Forum, 
Washington, DC, December 8, 1999). Washington, DC: Brookings Institution. Available: 
http://www.brook.edu/comm/transcripts/19991208.htm 
Back 



Gueron, J. (1999). Can we make education policy on the basis of evidence? What constitutes high quality 

education research and how can it be incorporated into policymaking? (transcript of a Brookings Press Forum, 

Washington, DC, December 8, 1999). Washington, DC: Brookings Institution. Available: 

http://www.brook.edu/comm/transcripts/19991208.htm 

Back (1st citationl 

Back (2nd citation 1 

Kaestle, C.F., Damon-Moore, L.C., Stedman, K., Tinsley, K., &Trollinger, W.V., Jr. (1991). Literacy in the 
United States. New Haven, CT: Yale University Press. 

Back 

Kamil, M.L., Langer, J.A., & Shanahan, T. (1985). Understanding research in reading and writing. Boston, MA: 
Allyn & Bacon. 

Back 

LeCompte, M.D., & Schensul, J.J. (1999). Designing and conducting ethnographic research. Walnut Creek, CA 

AltaMira. 

Back fist citation 1 
Back (2nd citation') 

Shanahan, T. (1999, August). The National Reading Panel: Using research to create more literate students. 

Reading Online. Available: http://www.readinaonline.ora/critical/shanahan/panel.html 

Back 



About the Author 

Victoria Purcell-Gates is a professor of literacy at Michigan State University (304 Erickson Hall, Department of 
Teacher Education, East Lansing, MI 48824, USA; e-mail vpgates@msu.edu ). Her research spans the 
methodological continuum from ethnography to experimental, and she has published widely in a variety of 
academic journals. Her book Other People's Words (Harvard University Press, 1995), based on an 
ethnographic study of a nonliterate family, won the 1996 University of Louisville Grawemeyer Award for 
Education for outstanding contribution to education and to "making the world a better place." Her most recent 
book. Now We Read, We See, We Speak ( Lawrence Erlbaum Associates. 2000), co-written with Robin 
Waterman, is an ethnography of a Freirean-based women's literacy class in El Salvador. She and coresearchet 
Nell K. Duke are currently working on an experimental study of explicit teaching of science genre features in 
second and third grades, funded by the National Science Foundation and the Interagency Education Research 
Initiative. 



Reading Online, www.readingonline.org 
Posted July 2000 

© 2000 International Reading Association, Inc. 



ERIC 



ISSN 1096-1232 



I 



5 



7 




U.S. Department of Education 

Office of Educational Research and Improvement (OERI) 
National Library of Education (NLEj 
Educational Resources Information Center (ERIC) 




NOTICE 



REPRODTJCTfON BASfS 




This document is covered by a signed “Reproduction Release 
(Blanket) form (on file within the ERIC system), encompassing all 
or classes of documents from its source organization and, therefore, 
does not require a “Specific Document” Release form. 




This document is Federally-funded, or carries its own permission to 
reproduce, or is otherwise in the public domain and, therefore, may 
be reproduced by ERIC without a signed Reproduction Release form 
(either “Specific Document” or “Blanket”). 




EFF-089 (9/97) 




